Configuration control in multiprocessors

ABSTRACT

THE PRESENT MULTIPROCESSOR APPARATUS INCLUDES, IN EFFECT, A DISTRIBUTED INTERCONNECTION SYSTEM SUCH THAT FAILURE OF A PORTION PF THE INTERCONNECTION SYSTEM DOES NOT COMPLETELY DISABLE THE APPARATUS. EACH ELEMENT-OF A PLURALITY OF COMPUTING ELEMENTS, A PLURALITY OF STORAGE ELEMENTS, AND A PLURALITY OF OTHER INFORMATION PROCESSING ELEMENTS OF THE TOTAL MULTIPROCESSOR APPARARUS-IS EQUIPPED WITH AN INDIVIDUAL CONFIGURATION CONTROL REGGISTER FOR SELECTIVELY CONTROLLING THE FLOW OF INFORMATION BETWEEN THE RESPECTIVE ELEMENTS AND OTHER ELEMENTS OF THE APPARATUS. FOR CONTROLLING THE INTERCONNECTION SYSTEM REPRESENTED BY THE CONFIGURATION CONTROL REGISTERS, WITH THE REDUNDANCY NECESSARY FOR RELIABLE &#39;&#39;&#39;&#39;FAIL-SAFE&#39;&#39;&#39;&#39; OPERATION OF THE SYSTEM, EACH OF A PLURALITY OF COMPUTING ELEMENTS IS PROVIDED WITH MEANS FOR INDEPENDENTY DEVELOPING, SELECTING AND CONDITIONING SIGNALS AND WITH MEANS FOR BROADCASTING THESE SIGNALS TO ALL ELEMENTS OF THE APPARATUS, INCLUDING THE ORIGINATING ELEMENT. THE SELECTING SIGNALS ARE ULTIZED AT THE RECEIVING ELEMENTS TO SELECTIVELY LIMIT APPLICATION OF THE CONDITIONING SIGNALS ONLY TO CONFIGURATION CONTROL REGISTERS OF ELEMENTS DESIGNATED BY THE SELECTING SIGNALS, SUCH ELEMENTS HAVING BEEN PREDETERMINED BY THE ORIGINATING COMPUTING ELEMENT. THE CONDITIONING SIGNALS SERVE TO CONDITION THE CONTROL REGISTERS OF THE SELECTED ELEMENTS TO DESIRED STATES OF INFORMATION FLOW CONTROL ALSO PREDETERMINED BY THE ORIGINATING COMPUTING ELEMENT.

July 24, 1973 s, STAFFORD EI'AL Re. 27,703

CONFIGURATION CONTROL IN MULTIPROCESSORS Original Filed June 2. 1965 12Sheets-Sheet 1 mhzmiu fi 02:31:90 muff-.0

WNW

328: 4 Q: 2 a 5 L 35 225E Jo EEO Jo 6 why-4a mm IFO 3: Pzwzud 052K200INVENTORS THDMAS S. STAFFDkD DONALD L. BURNSTINE GERARD T. PAUL JOHN R,ROGASM ATTDRNEY w bi 32% W 2 JQ; Q

mt K J: mofilzo July 24, 1973 "r. s. STAFFORD ETAL Re. 27,703

CONFIGURATION CONTROL IN MULTIPROCESSORS Original Filed June 3. 1965 12Sheets-Sheet 5 FIG. 6 COR CE smus 90 90 E S60 STORAGE COMPUTER (5) (SC)P ELEMENT P SE ELEMENTP 100E P (SE) (CE) fi/i/ V 121254H23456782EHOH|212543////(1254 CCR SE 7'// A s so P P CE P IOCE P IM24234 1 2/12545 1254 FIG. 8 CCR/IOCE /l s 50 P 5;: P SE cE P P CCR/TCUJuly 24, 1973 T. s. STAFFORD ErAL Re. 27,703

CONFIGURATION CDNTROL IN MULTIPROCESSORS Original Filed June 2, 1965 12Sheets-Sheet 7 ELEMENT FIG H RECONFIG. RESPONSE llFimiw CM/CEg/ AND TESTSWITCH OFF I0 P NF FQ 4 FROM VEi W SCON(T0 LINE 120,

11 PROGRAM CONTROLS\ TEST,

121 (FIGJO g0 UNF OFF MATCHL AND \NTE RRU PT CONTROLS ELC FROM OTHERELEMENTS July 24, 1973 "r. s. STAFFORD ETAL Re. 27,703

CONFIGURATION CONTROL IN MULTIPROCESSORS Original Filed June 2, 1965 12Sheets-Sheet 8 aecomnc. RESPONSE {SE v H v 1 0R M50 T FIG. 12

I 520 OR cm/ce L 30 1 SM/CE1IBIT s5 I CCR/SE1/BiT s0 M mo v ANDS 0/65SM/CEg/BIT SE1 6 5 36 CCR/SEq/BIT 50 AND ANDS 7 502 50s SMICE3/EIT as,as CCR/SE1/BHSC5 AND AN DS swam/an s5 "N306 307 CCR/SEq/BIT SE4 SE1 TESTLATCH OFF AND SE4 TEST LATCH 0N SE1 KEYS 1 CR8 cue/s5 s so P P c? P 10 P:[2 M21514 A1 //2 /1i2I5{4l3p 1M5 [L lgemsr 4 cmcun AND an TEST swncu onSE TET wr F L ccR/sE, OUTPUT T0 SE1 RECEIVING ems SE1 TEST LATCH ION OFFJuly 24, 1913 T. S. STAFFORD ET AL CONFIGURATION CONTROL INMULTIPROCESSORS Original Filed June 2, 1965 soon m 4 I I L! RECUNFIC.RESPONSE ([01 2 12 Sheets-Shea t 9 R F l G. i 3

OR CHI/(3E1 SM/CE an 10 1 1 I 36 CCR 10 mm 50 s AND ANDS W CE 2 SM 05 an6 56 CCR 10 mm AND ANDS W05 5 F.- sn/ce /en 10 8 4 CUR/.[DHBII 505 AND VANDS f as SH/CE4/BIT I01 CCR/[01/BlT 504 AND 7 ANDS 10, TEST mow OFF 10TEST men ON 56 1o 1 ms I 0 RS "56* T 00 my l- COR/l0 1 s 50 P SE P SF.if P a 1 z 1 i 213 [4 1 1M5 14 5|sl1 [a 2 9W1 M1 z {5M5 97 4 101 32 4BYPASS 4 51 52 cmcun 5 1 T I won ON AND 0 E6 8 10 15s? swncn OFF ICUR/[0g OUTPUT 10: TEST man {on on y 973 T. s. STAFFORD E Re. 27,703

CONFIGURATION CONTROL IN MIILTIPROCESSORS Original Filed June 2, 1965 12Sheets-Sheet 1O FIG. 14 INTERNAL OPERATIONS PERFORMED BY CE,UNDER SE PRGCONTROL, IN CONNECTION WITH RECONFIGURATION INITIAI'E PROGRAMINTERRUPTION IN RESPONSE To ELC 0R SUPERVISOR-T CALL INSTRUCTIONENCOUNTERED TR CURRENT PROGRAM COLLECT CURRENT PROGRAM STATUS WORD IPSWIFROM LOCAL BUFFER 226 REGISTERS AND STORE AT UNIQUE LOCATION IN APREDETERMINED SE FETcR RER PSW FROM ANOTHER UNIQUE LOCATION IN sEACCORDING T0 CLASS OF INTERRUPTION DISTRIBUTE REw PSW T0 LOCAL BUFFERREGISTERS AND PROCEED TO FIRST INSTRUCTION OF NEW PROGRAM AT SE ADDRESSREFERERcER IN 228 NEW Psw 229\ INVESTIGATE ELEMENT ccR STATES VIA STATUSTABLE sToREn AT PREDETERMINED ADDRESSES IN SE INCOMPLETE PROCEED"CONFIGURATION COMPLETE EXIT FETCH scoR INSTRUCTION: RI R2 (240 FROMSELECTED ADDRESS IN SE TO LOCAL BUFFERS; VERIFY SUPV. MODE BIT IN PSW;CHECK CM IN RI FOR PERMISSIBLE SC BIT CONFIGURATION; CHECK OWN OCR/CESTATUS BITS AND OWN CE TEST SWITCH; SET SCON LATCH I SIMULTANEOUSLYTRANSFER CM FROM R1 AND SM/BIT CE FROM R2 TO OWN CCR RECEIVING GATES IFOWN STATUS BITS ARE OD OR II, AND TO ALL OTHER CCR RECEIVING GATES IF,IN

ADDITION, OWN CE TEST SWITCH IS OFF I I REsET SM BITS IN R2 INACCORDANCE WITH ELEMENT RECONFICURATION RESPONSE U k UPDATE STATUS BITSDE RESPONDING ELEMENTS IN STATUS 233 TABLE July 24, 1973 T. s. STAFFORDETAL 27.703

CONFIGURATION CONTROL IN MULTIPROCESSORS Original Filed June 2, 1.965 12Sheets-Sheet 1:

FIG. 16

ELEMENT CCR/BIT s CCR/BIT s PAN; 1 1

TCU1 1 1 United States Patent Oflice Re. 27,703 Reissuecl July 24, 197327,703 CONFIGURATION CONTROL IN MULTIPROCESSORS Thomas S. Stafford,Wappingers Falls, Donald C. Bumstine, Hyde Park, Gerard T. Paul,Poughkeepsie, and John R. Rogaski, Woodstock, N.Y., by InternationalBusiness Machines Corp., Armonk. N.Y., assignee Original No. 3,386,082,dated May 28, 1968, Ser. No. 460,776, June 2, 1965. Application forreissue Sept. 5, 1972, Ser. No. 288,182

Int. Cl. G061 9/00, 11/00 US. Cl. 340-1725 18 Claims Matter enclosed inheavy brackets appears in the original patent but forms no part of thisreissue spec fication; matter printed in italics indicates the additionsmade by reissue.

ABSTRACT OF THE DISCLOSURE The present multiprocessor apparatusincludes, in eflect, a distributed interconnection system such thatfailure of a portion of the interconnection system does not completelydisable the apparatus. Each element-of a plurality of computingelements, a plurality of storage elements, and a plurality of otherinformation processing elements of the total multiprocessor apparatus-isequipped with an individual Configuration Control Register forselectively controlling the flow of information between the respectiveelement and other elements of the apparatus. For controlling theinterconnection system represented by the Configuration ControlRegisters, with the redundancy necessary for reliable fail-safeoperation of the system, each of a plurality of computing elements isprovided with means for independently developing, selecting andconditioning signals and with means for broadcasting these signals toall elements of the apparatus, including the originating element. Theselecting signals are utilized at the receiving elements to selectivelylimit application of the conditioning signals only to ConfigurationControl Registers of elements designated by the selecting signals, suchelements having been predetermined by the originating computing element.The conditioning signals serve to condition the Control Registers of theselected elements to desired states of information flow control alsopredetermined by the originating computing element.

The invention hereof pertains to the control of inter connectionsbetween independent elements of a multiprocessing complex. Inparticular, it pertains to a system of connection controls which can beused to reliably maintain continuity of data processing operations andeflicient distribution of element work load assignments within amultiprocessing complex in a variety of problem situations.

In certain real-time data processing situations, for example in theprocessing of air traflic control data, it is necessary to maintainsubstantial continuity of operation in order to avoid loss of data whencircuit failures or other causes of processing bottlenecks are imminentor have actually occurred. It is conventional in this regard to provideduplicate system elements which may be maintained available, on astandby or other basis, to be switched into an active processing systemas a substitute for a failing element or to back up an element which isbeing overloaded with too many tasks. The weak link in this procedurehas been the means used to accomplish the switching in of the back-upelement and, if need be, the disconnection of the failing element.Hitherto, the practice has been to provide a centralized arrangement ofmaster switching controls which, in response to signals indicating anelement failure, or other condition requiring attention can act toswitch in back-up elements so as to reorganize the processing system, orsystems, then in operation. One problem associated with this form ofcontrol is that the master switching controls themselves are susceptibleto failures capable of interrupting operations and causing irreparableloss of both data and data processing time.

It is, of course, possible to provide for manual disconnection of themaster controls and the, inter-element switches when a failure occurs.However, when time is of the essence, manual control is inadequate sinceseconds, or even minutes, might elapse between the time of occurrence ofa failure and the disconnection of the master controls, during whichtime thousands of misrouted data transfers may occur wtih consequentloss of much data and disruption of data processing operations currentlyin progress.

Accordingly, an object hereof is to provide an improved system forcontrolling the configuration of interconnections between elements of amultiprocessing complex.

Another object is to provide a system for controlling interconnectionsbetween elements of a multiprocessing complex which can be adapted toselectively reorganize interconnections between elements to maintainprocessing operations within the complex at a prescribed level ofefiiciency, continuity and reliability.

Another object is to provide a system for efiiciently controllinginterconnections between elements of a multiprocessing complex which canbe expected to effectively maintain continuity of data processingoperation during ordinary system failures.

Yet another object is to provide a system for etficiently controllinginterconnections between elements of a multiprocessing complex which canbe expected to reliably maintain continuity of processing operationduring malfunctioning of any element of the complex including an elementwhich can exercise supervisory control over all interconnections withinthe complex.

Another object is to provide a system for efficiently controllinginterconnections between elements of a multiprocessing system includingcheck features which preclude erroneous initiation of a change ininterconnections or undesirable acquiescence by an element in aproffered change.

A feature of the subject configuration control system resides in theprovision, within each of a plurality of computing elements of amultiprocessing complex, of means capable, under prescribed conditions,of effecting changes in the states of receptiveness of any elements ofthe complex signals transmitted thereto from other elements of thecomplex. Subject to certain restrictions any computing element cantransmit configuration control signals which are selectively received bythe other elements of the complex and determine the states ofreceptiveness of the receiving elements. It is thus unnecessary todisconnect power from a faulty element to render other elementsinsensitive to it and by appropriately configuring states ofreceptiveness, the processing tasks of a faulty element can bereassigned to a health standby element, or added to the duties of anactively functioning element. Since such configuring can be accomplishedautomatically in response to a fault indicating signal it is possible tomaintain continuity of processing when a failure occurs. It is evenpossible to allow a faulty element to continue to transmit signalsexternally while it is undergoing diagnostic testing or othermaintenance procedures without risk of disrupting data processingoperations currently in progress. of perhaps equal significance,processing bottlenecks, due to assignment of too many tasks to onecomputing element can be anticipated and another computing element canbe programmed to automatically step in and reconfigure the system so asto relieve the overloaded element while the latter remains free tocontinue its necessary activities without interruption.

Another feature herein is that internal to each element of the complexthere is provided a configuration control register (abbreviated CCR)having output control consections to gates which receive and passsignals transmitted by other elements. Thus, the receptiveness of anelement to transmissions of other elements is a function of the presentstate of its CCR. The inputs of any CCR, subject to the output states ofcertain bits therein, are accessible to each computing element of thecomplex. Thus the content of any CCR, including a computing element CCR,may be selectively changed by any one of the :omputing elements.

Another feature is that each element can transmit an :lement checksignal (ELC) in response to which the :omputing elements can initiate areconfiguration program leading to modification of selected CCRs andconsequent reorganization of the complex. In response to an ELC thosecomputing elements enabled by bits in their re- ;pective CCRs to receivefrom the element transmitting :he ELC select one of their number toissue a program of reconfiguration signals. These signals establish newini'ormation settings in selected CCRs, including, if need Je, the CCRassociated with the element issuing the ELC tignal. The issuance of anELC signal may be conditioned 1pon the results of parity checks, powersupply checks, ogic checks, work load checks or checks of any other:lement condition which is sufficiently pertinent to sysem operations towarrant consideration of a reconfiguraion program.

The program instruction executed by a computing elenent to reorganizeelement system assignments within he complex-i.e. to modify selectedCCRs--is referred vo herein as SCON (abbreviation for SetConfiguraion"). Programs containing such instructions are strategi-:ally located in a plurality of storage elements of the com lex on aredunant basis so that failure of any one stortge element will notprevent reconfiguration of the com- :lex.

In response to an element check or other condition callng for possiblereconfiguration of a system, one of the :nabled computing elementsdetermines the condition of he complex, and if circumstances warrantfurther action, t retrieves one or more appropriate SCON instructions'rom an associated storage element and executes them. Jnder thedirection of the selected SCON instruction, the :omputing element issuesa reconfiguration signal, known B a configuration mask (CM), and anelement selection :ignal, or selection mask (SM). The CM signals areseectively accepted as new CCR entries by the elements iesignated by SM.

The foregoing and other objects and features of the nvention will bemore fully understood and appreciated lpon consideration of thefollowing description thereof .aken with reference to the accompanyingdrawings wherein:

FIGS. 1-4, when arranged according to the plan shown 11 FIG. 5,constitute a schematic illustration on a gen- :ral level of amultiprocessing complex including a coniguration control systemorganized in accordance with be present teachings;

FIGS. 6-9, inclusive, provide diagrammatic illustraions of the format ofthe configuration mask informaion stored in the configuration controlregisters (CCRs) tssociated with the different types of elements shownin he complex of FIGS. l-4;

FIG. 10 is a schematic of a representative computing :lementconfiguration control register (CCR/CB and iccess controls therefor;

FIG. 11 is a schematic drawing illustrating parts of a epresentativecomputing element CB which can conrol the acquisition and execution ofSCON instructions For reorganizing the multiprocessing complex hereof;

FIG. 12 is a schematic illustration of pertinent parts of arepresentative storage element configuration control register (CCR/SEand associated access controls therefor:

FIG. 13 is a schematic of pertinent parts of a representativeinput/output control element configuration control register (CCR/IOCEand associated access controls;

FIG. 14 is a schematic flow diagram of a sequence of operations executedinternally by a computing element in connection with the handling of aSCON instruction;

FIG. 15 contains a table which defines the operational capabilities ofeach element with reference to certain status bits (S) contained in theCCR internal to the element;

FIG. 16 illustrates a typical stored status table listing the currentstatus bits of all element CCRs of a complex. This table may be used asa stored reference by a CE in connection with the Instruction Fetchingroutine to determine its selection of a SCON instruction for execution.

GENERAL DESCRIPTION As shown in FIGS. 1-4, a multiprocessing complex ofthe type under consideration herein typically may include a plurality ofcomputing elements (CE), shown at 1 and 2 in FIG. 1, and may include aplurality of other specialized elements, such as storage elements (SE),shown at 3 and 4 in FIG. 2, input-output control elements (IOCE or IO)shown at 5 and 6 in FIG. 3, and peripheral adapter elements 7, 8 (FIG.4).

The adapter elements are each connected to a plurality of peripheraldevices such as tape stores, communication terminals, printing devices,and the like. The tape store adapter units are hereinafter designated bythe symbol TCU (abbreviation for tape control unit) and other peripheraldevice adapters are designated by the letters PAM (abbreviation forperipheral adapter module).

As may be inferred from the descriptive terminology used above, CEsperform processing operations on internal programs and informationsupplied by SEs and IOCEs provide a link for exchanging informationbetween the SES and external input and output devices via the additionalbufl'ering afforded by the adapter elements.

Each element is provided with a configuration control register 10(abbreviated CCR), two sets of receiving gates 11, 12 and other internalcomponents 13 collectively denoted other parts. These other parts mayinclude individual power supplies. To simplify FIGS. 1-4, only theinternal organizations of representative elements, CB SE IOCE and TCUare shown. It should be understood that the internal organizations ofother elements in each category would be similar to those of therepresentative elements.

The receiving gates 11 control the introduction of new informationsettings into the respective CCRs, and the receiving gates 12 controlthe reception of other information by the other internal circuits 13 ofthe respective elements. The receiving gates 11 and 12 of each elementare controlled by various digit outputs of the associated CCR asindicated by lines 15, 16 and 17. Thus, all incoming information signalsproffered to an element can be selectively received or rejected by theelement in accordance with the contents of its associated CCR. Anexception to this is found in the IOCE elements. Information profieredto an IOCE 5, 6 by a peripheral adapter unit 7, 8 is permitted to bypassthe receiving gates 12 controlled by CCR/IOCE. This is because there areother internal receiving gates in each IOCE, not controlled by CCR/IOCE,which nevertheless prevent unauthorized reception of informationtransmitted from an adapter.

The other parts" 13 of each SE, 3, 4, include a fast access store, suchas a magnetic core matrix memory, and access controls for reading andwriting information relative thereto.

The other parts 13 of each CE include computing and control circuits,for operating upon stored program instructions and data stored in an SE,for executing SC'ON instructions and selectively issuing configurationcontrol information to the CCRs, and for handling the acknowledgingresponses of elements intended to have their CCRs reconfigured by suchinformation.

The other parts of each IOCE include one or more channel systems forselectively conveying information bidirectionally between peripheraldevice adapters (TCU, PAM) and SEs, according to the needs of the CEs.

Data and control signals not directly connected with configurationcontrol are exchanged between a CE and and SE via group bus 20, one ofthe tributary buses 21 (FIG. 1) or 22 (FIG. 2) feeding into bus 20, andone of the branch buses 23 (FIG. 2) or 24 (FIG. 1) extending from thegroup bus.

Data and control signals not directly connected with configurationcontrol are handled between CE and IOCE units via group bus 25 (FIGS. 1and 3), one of the tributary buses 26 (*FIG. I) or 27 (FIG. 3) feedinginto group bus 25, and one of the branch buses 28 (FIG. 3) or 29(FIG. 1) respectively, extending out of the group bus.

Data and control signals not directly connected with configurationcontrol are handled between SE and IOCE units via group bus 35, one ofthe tributary buses 36 (FIG. 2) or 37 (FIG. 3), and one of therespective branch buses 38 (FIG. 3) or 39 (FIG. 2).

One other path for the handling of data and control signals is thatbetween the IOCE units (FIG. 3) and peripheral adapter units (FIG. 4).It includes paths defined by group bus 45, one of the tributary buses 46(FIG. 3) or 47 (FIG. 4), and a respective branch bus 48 (FIG. 4) or 49(FIG. 3).

Each element of a group of elements having tributary buses feeding agroup bus can transmit signals selectively to any element of a grouphaving branch buses extending from the same group bus. Thus, any of them computing elements, CE can selectively send information to and receiveinformation from any one of the n storage elements SE via group bus 20and appropriate ones of the tributary and branch buses 21-24. Hereafter,the foregoing connections will be referred to as data paths todistinguish them from configuration control information handlingconnections between CEs and CCRs, referred to as configuration controlpaths. The latter paths are described in greater detail below.

Means not shown are provided for resolving conflicts created byconflicting calls for service by two or more elements from a singleelement; for example, simultaneous calls by two or more CEs for accessto the same SE. Such means act to assign priority to a unique one of thecalling elements according to a predetermined plan. Since this priorityassignment plan is not pertinent to the configuration control system ofthe present invention it is deemed unnecessary to provide furtherdetails thereof in this discussion.

The following brief example will sufiice to complete the description ofthe data paths. Suppose CB is engaged in executing a program stored inSE and wishes to retrieve its next instruction from an InstructionAddress IA in SE CB raises a call line, in a path 21, 20, 23, between CEand SE and transmits a representation of the address IA, along otherlines in the same path. At some later time SE runs through anon-destructive read/ regenerate cycle of access to internal addresslocation IA. Upon readout of the contents of IA, SE raises a responseline in the data path 22, 20, 24 between SE and CB while at the sametime transmitting the contents of IA to a special buffer register in CBvia other lines in the same path. The response signal is used by CE; toreset its call signal and the exchange is completed.

Signals participating in the establishment of new configuration controlsettings in CCR register are handled through a configuration control bussystem including a main group bus 60 (FIG. 1), subgroup buses 61 (FIG.2), 62 (FIG. 3), 63 (FIG. 4), and a plurality of tributary and branchbuses and lines as described below. While any element may initiate arequest (ELC) for action which may result in a new CCR setting, only aCE (FIG. 1) can issue the actual CCR setting information, hereinaftertermed configuration mask (abbreviated CM). Thus, each CE is providedwith a CM output bus such as the bus 71 extending from CB (FIG. l), andeach element CCR is provided with a CM input bus 72 for carrying a CMfrom any CE to the receiving gates 11 of that CCR.

An abbreviated form of notation is employed herein to characterize theinformation units pertinent to a reconfiguration process. In generalthis notation has the form: A/B/C, where A represents the informationunit, B represents the issuing source, and C specifies particular bitsin A. Thus, the notation employed for example to designate particularbits a, b of a 36-bit CM word issued by CB would be CM/CE /bits a, b.

As indicated above, a CM word issued by a CB unit via a CM output bus,such as 71 (FIG. 1) would be applied in parallel to the CCR receivinggates 11 of all elements in FIGS. l-4 via respective CM branch buses 72extending in parallel to all the inputs of such gates. Thus, thenotation CM/CE at 72 indicates that all CM/CE outputs extend as inputsvia 72.

Those elements which are intended to receive an issued CM are designatedby corresponding bits in a coincidentally issued Selection Mask word(SM). The latter is provided by the CE issuing CM on its associatedSM/CE output bus 73. Thus, the gates 11 of CCR/CB (FIG. 1), CCR/SE (FIG.2), CCR/IOCE, (FIG. 3), and CCR/TCU (FIG. 4), are respectively selectedby the conditions of bits designated bit CE bit SE bit I0 and bit TCU inan SM word. 'Since such SM words may issue from any of in source CEs, aseparate SM bit line is extended from the bus 73 of each CE to eachgroup of element CCR gates 11. Thus, the lines 74 extending to the gates11 of CCR/CE in FIG. 1 can convey a CB selecting bit of an SM wordissued by any CE (including CB to the receiving gates 11 of CE,, asindicated by the notation SM/CE /bits CB Similarly, the lines 74extending to SE (FIG. 2) convey SE selecting bits of SM words to the SE,receiving gates 11 as indicated by the notation SM/CE /bits SE Likewisein FIG. 3, the notation SM/CE /bits I0 adjacent bus 74 indicates thatthe I0 selecting bits of all SM words are conveyed to the gates 11 ofIOCE Finally, in FIG. 4, SM/CE /bits TCU indicates that a TCU; selectingbit in each SM word is conveyed thereby to gates 11 of TCU It should beunderstood that the SM selecting bits are binary-valued signals which indifferent binary states will tend to actuate or not actuate respectivegates 11. It may further be inferred by noting the lines 15 and 16extending from each CCR to the gates 11 that each CCR can selectivelyaccept or reject a proffered CM (Le. a CM accompanied by an activeselecting bit of an SM on a line 74) in accordance with the informationcurrently stored in the CCR. The lines 15 issue from a pair of statusbit stages (S S of each CCR, the combined states of which determine fourelement states: 00, 01, 10, and 11. The lines 16 issued from individualsupervisory control bit stages (SC of the respective CCR, and correspondindividually to the different CE elements of the complex. Signals online 16 control acceptance of incoming CMs proffered by SM selectingbits of respective CE units. The status bits S determine whether theassociated CCR is in a state permitting acceptance of a proffered CM,and if the state is such a permissive one, the control bits SCselectively determine the CEs from which a CM may be accepted (i.e. theelement selecting bits of SM to which the gates 11 may react). Anelement accepting reconfiguration signals its acceptance of theproifered CM to the issuing CE via an individual response line 75.Failure to receive such a response from any element designated by SM istaken as an indication of program error, or other failure, by theissuing CE.

Over-riding control over the S and SC bits in each element CCR isafforded by reset (R) input lines 76 which are coupled to reset outputlines 77 issuing from the CEs (FIG. 1) by manual reset controls 78, andSC bypass circuits which are not shown in FIGS. l-4. The latter circuitswhich are considered in detail in connection with the discussion hereinof FIG. 10 are effective to force all SC bit outputs of CCR to the 1condition when all SC bits are coincidentally set to 0. The effect thusproduced is the same as if all SC bit states in the CCR had been setto 1. Were it not for this, if all SC bits in CCR became coincidentallyset to 0 due to some faulty condition the CCR of the associated elementwould become isolated from further reconfiguration control.

Each element is further provided with an Element Check signalling line(ELC) 79 by means of which the existence therein of an error or othercondition requiring attention is made known to the CEs. Recall that theCE states (S) are so maintained that at least two CEs will be able torespond to the ELC. Thus invariably one CE will respond to the ELC,execute SCON, assemble CM and SM, and thereby reconfigure the complex.In response to an ELC signal which is a pulse of short duration, subjectto certain checks and priority resolution procedure described hereafter,one of the enabled CEs, other than a CE originating an ELC, will performa program routine to evaluate the cause of the error or failure whichgave rise to the ELC siganl. This one CE will then assemble and forwardSM and CM words under control of SCON instructions through the bussystem described above. These words condition the selected CCRs andthereby reconfigure the complex. Assuming that the SCON instructions areappropriately programmed the data processing operations in progressprior to the ELC may be continued without significant interruption.

The response and ELC output lines 75 and 79, of all elements areextended in parallel via branch buses 80 (FIG. 1) to the receiving gates12 of each CE.

The general effects characteristic of a reconfiguration process asexecuted by the apparatus shown in FIGS. l-4 may now be summarized asfollows:

First, one or more CEs are made aware by an ELC or an internalcondition, such as a decoded program instruction or manual switchsetting, of a problem condition. After executing several internal checksto determine whether they can legitimately issue reconfigurationinformation, the CEs attempt to gain access to a predetermined one ofseveral SEs containing information pertient to reconfiguration. Byexecuting a priority resolution procedure these CEs enable a unique oneof their number to address the SE and withdraw therefrom a status tablecontaining a listing of the status bits (S) currently stored in all CCRsof the complex. Utilizing this status intelligence the enabled CEexchanges other program information with an associated SE to determine aconfiguration of the complex suitable for maintaining continuedprocessing. Branch decisions in this program guide the controls of theenabled CE to selectively address and execute one or more SCON (SetConfiguration) instructions in SE appropriate to the circumstances. EachSCON instruction is retrieved and used by the enabled CE to control acorresponding internal subroutine during which appropriate CM and SMwords are assembled in CB registers. Thereupon, the CM word is broadcasttowards all elements via bus 60 and proffered to selected elementsaccording to the coincident states of corresponding selecting bits ofSM. Those elements designated by SM which are permitted by the S and SCbits in their CCRs to accept the proffered CM do so and acknowledgetheir acceptance by a signal sent to the issuing CE via the responseline 75. The response signal is utilized by means not shown in theissuing CE to reset the corresponding SM selecting bit in the CEregister preloaded with the SM word.

When the mask issuing CE determines, by internal controls furtherconsidered herein, that all elements designated for reconfiguration byone or more SMs have accepted the correspondingly proffered CM, andtherefore verified that the complex has been suitably reconfigred, itmay terminate its execution of SCON instructions and resume operation aspart of a processing subsystem within the complex, retrieving its nextprogram instruction from an SE to which it is configured by acorresponding SE bit in its CCR.

It will be noted that the output lines 17 of each CCR control thereceiving gates 12, of the associated elements, through which allinformation, other than configuration control information, or adapterinformation proffered to an IOCE, reach the internal parts 13 of theassociated element. Thus, it will be understood that outputs of any CCRconnecting to respective lines 17 control reception of informationtransmitted to the associated element from other elements and therebythe CCR contents in effect determine all signal flow between elements ofthe complex. A feature of this reception control is that a faultyelement, even if it should become isolated from reconfiguration controland continue to transmit erroneous information, can still be cut offfrom all "healthy" elements of the complex by reconfiguration; i.e. bychanging a bit in each healthy elements CCR.

CCR FORMATS Considering FIGS. 6l3, with reference to the complex shownin FIGS. l-4, organizational details of the subject configurationcontrol system may now be explained FIGS. 69 show the relative formatsof the control information stored in the respective CCRs of the CE, SE,10, and adapter elements. The information in each CCR consists ofselected parts of a basic 36-bit control word or CM format.

In the particular embodiment of a multiprocessing complex describedherein, there are provided four CEs, a maximum of 12 SEs, and a maximumof 3 IOCEs, (Le. m=4, n=12, and 13:3). Accordingly, each CCR containsfour supervisory control bit stages (SC the outputs of which controlacceptance of a new CM proffered by corresponding ones of CE Each CCRalso contains a pair of status bit stages (S and from one to four paritybit stages (P Each parity bit stage holds the anticipated correct parityof the 8 consecutive bits to the left thereof in the CCR and may becompared to a computed parity as a check on the correctness of those8-bits.

In CCR/CE (FIG. 6) there are 12 bits stages, designated CCR/CE /bits SE,which have outputs controlling the receipt by the associated CE ofintelligence forwarded by respective storage elements SE of the complex.There are also 4 bit stages CE controlling receipt by the particular CEof intelligence conveyed from correspondingly numbered CEs, and bitstages IOCE E (or 10 for controlling acceptance by the particular CE ofsignals originated by corresponding IOCEs. The seven blank bit positions90, distinguished by a pattern of parallel slanted lines, are consideredspare or dont care positions since the outputs therefrom do not controlreceiving gates.

CCR/SE (FIG. 7), in addition to S, SC and P bit stages contain bitstages CE, and 10 corresponding to the CE and I0 bit stages in CCR/CE(FIG. 6) for controlling acceptance by SE of signals conveyed fromreceptive CEs and IOCEs. All other bit positions are spare (unused).Since the eight positions 91 to the left of P are blank all of the ninestages 91 and P are unnecessary and may be omitted.

CCR/IOCE (FIG. 8) in addition to bit stages denoted S, SC and P, containbit stages SE and CE, corresponding to correspondingly designated SE andCE bit stages in CCR/CE, for controlling acceptance by IOCE of signalsconveyed from respective SEs and CEs. All other bit positions are spare.Since stage P and the associated eight stages 92 are not used, all nineof the stages may be omitted.

The CCR of each peripheral adapter, for example, CCR/TCU (FIG. 9), inaddition to S, SC and P bit stages contains bit stages IO in positionscorresponding to those held by the IO bits in CCR/CE, for controllingacceptance by the respective TCU of signals supplied by IOCErespectively. All other stages contain unused or spare bits. Since thestages 93 and 94 respectively associated with P and P are unused, alleighteen of these stages are unnecessary and may be omitted.

Considering the CCR formats (FIGS. 6-9) in relation to the associatedelements of the complex as characterized in FIGS. l-4, a number ofpertinent observations may be made. An element (for example an SE) maybe prevented by a CE bit in its CCR from receiving ordinarycommunications from a particular CE and yet be permitted by an SC bit inits CCR to accept a CM proffered by the same CE. Secondly, since theCCRs of all elements have similar control information formats-forexample, the status and SC bits invariably occupy the first six registerpositions in all CCR registers, and CE bits, where available, areinvariably located in the last four places of a group associated withparity stage P -a single pair of CM and SM words can be used to set aplurality of CCRs when circumstances permit (e.g. when the associatedelements are grouped together in a subsystem). Thus, the network ofgates and the programming required for entering information into a CCRmay be simplified. Finally, it is repeated that while the adapterregisters, such as CCR/TCU, have 10 bits to control reception of IOCEoriginated information, the IOCE CCRs do not contain adapter bits forcontrolling information flow in the reverse direction. The reason forthis is that the ordinary reciprocal controls between IOCE and adapterunits are so arranged that if one cannot receive a communication fromthe other, it cannot provide a key sequence of responses necessary toactuate the other. It is, therefore, sufficient to provide CCR controlin one direction only.

CCR/CE AND SETTING LOGIC Referring to FIG. 10, the configuration controlregister CCR/CB of a representative computing element CH and thecircuits controlling access thereto which are substantially identical ininternal organization to similarly functioning circuits in the other CEs(GE are seen to comprise detail features, as follows: For admitting newconfiguration control settings (CMs), proffered by any of the CEsincluding CE to the 36-stage register 100, which is the CCR/CBassociated with CE there are provided four AND circuits 101-104. Theoutputs of these control four respective groups of AND circuits 105-108.Each group of circuits 105-108 comprises 36 individual AND circuits forhandling the individual bits of a CM signal presented to the group.Groups 105 to 108 are coupled to receive CM signals originating atrespective CEs, CE, to CH via respective groups of input wires 110 to113. OR gates 115 combine respective outputs from each group of ANDcircuits 105-108 to provide a single 36-bit input path feeding intoregister 100.

AND circuits 101-104 are partially controlled by CE; bits in selectionmask signal words (SM) issued by respective CEs, CE As an additionalpartial control, signals corresponding to the outputs of bit stages SCof register 100, except under conditions noted below, are applied torespective ones of circuits 101-104. As a third partial control, SCONenabling signals applied via line 119 to AND circuit 101, and SCONenabling signals transmitted by other CEs are applied via line 120 tothe other three AND circuits 102-104. A fourth control input 121, toeach AND circuit 101-104, is a TEST LATCH OFF signal provided by CE,control circuits shown in FIG. 11. This line is disabled only when thevalues of both status bits, S and S in CCR/SE are coincidentally 0 whilea CB Test Switch (FIG. 11) is in an ON position, and it is restored toan enabled cond tion when the switch is thereafter placed in an OFFposition.

An additional group of 36 inputs to OR circuits 11.5 not previouslymentioned is supplied by lines 125. Signals on these lines correspond torespective manually keyed inputs 126 of 36 AND circuits indicatedgenerally at 127, when the latter are enabled by a control signal on acommon input line 129. Line 129 couples to the ON output of the TESTLATCH feeding line 121, so that the circuits 127 are enabled only whenthe line 121 becomes disabled; i.e. only when bits S and S in CCR/CB areboth 0 while the CE, Test Switch is ON. It is, therefore, clear that thecontents of register can be manually changed through action of keys orswitch levers only under certain predetermined status bit and testswitch conditions, this feature serving as an interlock or precautionarymeasure precluding accidental isolation of an element from CE, as aresult of accidental manipulation of a CCR/CB bit setting key.

The outputs of AND circuits 101-104 are also extended to OR circuit 133which couples to the CE configuration response line 75, also shown inFIG. 1. Upon application of a CE, selecting bit of any selecting mask SMto one of the AND circuits 101-104, and translation of a signal throughthe circuit, a response is conveyed via line 75 to the CE issuing theSM. This response is used to reset the CE, selecting bit in the registerholding SM whereby the issuing CE is informed that CE; has accepted aproffered CM in CCR/CB When energized, reset line 135 sets each of bitsSC to 1, and each status bit 5,, S to 0, so that if the CE, tcst switchis OFF, AND circuits 101-104 will all be enabled, permitting any CE toreconfigure CCR/CE.

A bypass circuit 137 translates the outputs of bit positions SC ofregister 100 to the output bus 138. This circuit includes four ORcircuits 139-142, effective to directly translate respective SC outputsof register 100, so long as one or more of these outputs are in the 1state, and an OR circuit 143 and complementing circuit 144 providing anindirect second input to OR circuits 139-142, which is active wheneverall bit stages SC in register 100 simultaneously contain zeroes. ANDcircuits 101- 104 cannot all be simultaneously disabled by CCR/CE bitsSC since the corresponding inputs are taken from the bypass circuit 137,and the four outputs of the latter invariably include a 1" signal. Thus,register 100 cannot be accidentally isolated from all CEs. As anadditional feature, the circuits 137 produce an all ones" output, upondetection of a CCR parity error, as suggested by inline 145 joining theoutput of complementing circuit Each of the other CEs, CH while notillustrated in detail should be understood to comprise CCR gates andcontrols corresponding in function and organization to those shown inFIG. 10 for CB except that the identitles of inputs to the circuitscorresponding to AND circuits 101-104 and 127 of CE, are different foreach of the CEs. For CB the AND circuits in the positions of ANDcircuits 101, 103 and 104 would be controlled jointly by CB TEST LATCHOFF and SCON signals origmating at other CEs, and individually by bits0E in SM/CE SM/CE and SM/CE, respectively, and by bits 8C S6 and SC,from respective stages of CR/CE2- The AND circuit in the position of AND102 would have inputs SCON/CB SM/CE /bit CB and CCR/CE /bit 8C The ANDcircuit corresponding to AND circuit 127 would be conditioned to pass CBdata key signals by CE: TEST LATCH ON, and the output of the OR circuitcorresponding to OR circuit 133 would represent the reconfigurationresponse of CE For CB the AND circuits corresponding to 101, 102 and 104would be controlled jointly by C5 TEST LATCH OFF and SCON signalsproduced by CE CB or CB and individually by bits CE, of SM/CE SM/CE andSM/CE respectively, and outputs of bit stages SC SC; and 5G,,respectively of COR/CB The inputs to the cir- :uit corresponding to ANDcircuit 103 would be SCON/ CB3, CF13, and The it! puts to the circuitcorresponding to AND circuit 127 would be CB TEST LATCH ON and CB DATAKEYS.

For CE, the counterparts of AND 101, 102 and 103 would have joint inputsCB TEST LATCH OFF and SCON (controlled by CE CB or CB as well as firstindividual inputs SM/cE /bit CB SM/ CE bit CE.,, and SMICE /bit CBrespectively, and second individual inputs CCR/CE /bit S CCR/CEq/bit Cand CCR/ CE /bit SC respectively. The counterparts of AND 104 would haveinputs SCON/CB SM/CE /bit C13,, and CCR/CEq/bit SC.,. The counterpart ofAND 137 would nave inputs CE, TEST LATCH ON and CE, DATA KEYS.

ISSUANCE OF SCON BY CE Consideration is next given with reference toFIGS. 11 and 14 to the other internal parts 160 of CE, and by analogy tocorrespondingly functioning parts in the other :Es. These includecircuits which participate in the acquiaition and execution of SCONinstructions and the collecion and distribution of CM and SM words toselected CCRs throughout the complex, with consequent reorganization ofthe associated elements within the complex. the parts 160 are groupedinto four main subdivisions iesignated receiving gates 161 (same asgates 12 in FIG. l), computing and parity checking circuits 162,interrupt :ontrols 163 and instruction decoding and other controls l64.The latter includes CE, Test Switch 165, other manial controls 166, andother electronic controls yet to be liscussed. The circuits 162 carryout computing and parity rhecking functions on information handledwithin 160 in- :luding checks on the information content of theconfiguation control register 0 (CCR/CH of FIG. 10. The :ircuits 164provide all of the sequential control signals .nd include sufficientlocal buffer storage registers to en- .ble CB to carry out its computingand data handling unctions effectively.

The other manual controls 166 include switches for :ontrolling powersources, single cycle operation, and Ither functions requiring manualcontrol. The controls 66 are effective only when a bistable test latch167 is in he ON state as indicated by a predetermined signal level nline 168. Latch 167 is set to the ON state by an active Iutput from ANDcircuit 169. The latter occurs only then test switch 165 is in the ONcondition and concidentally an enabling output is received from ANDircuit 170 in response to 0 indication from stages 8, and 2 of CCR/CE,(FIG. 10). Once in the ON state latch 67 can be reset to OFF only bysetting the test switch 65 to its OFF position. Thus, the bits S and 8;,can be aried for test purposes While the test latch remains in the )Nstate.

The OFF and ON outputs of test latch 167 also extend a the lines 121 and129 respectively shown in FIG. 10. Vhenever the condition of the testlatch is reversed from )N to OFF line 171 in FIG. 10 is energized toreset bit tates S and S of CCR/CB to 0, whereby CCR/CB rerainsaccessible to either manual reconfiguration by turnig the test switch165 back to ON, or by programmed :configuration as considered below.

In addition to the test latch there is provided a SCON SetConfiguration) enabling latch 175 which can be set an ON condition by anoutput from AND circuit 176. fircuit 176 is partially enabled by onesignal 177 of a roup of sequential microprogram signals schematically:presented at 178. These signals are derived during. a CB iicroprogramof operations culminating in the fetching to 013 of a SCON instructionfrom one of the working storages SE as described below. The signal 177is invariably followed by a signal 179 which establishes latch in theOFF or RESET state. In its ON state latch 175 provides a SCON enablingsignal to input line 119 of AND circuit 101 in FIG. 10. If, in addition,test latch 167 is OFF, a remote SCON enabling signal is translatedthrough AND circuit 180 to the receiving gates of all other elementCCRs. In each CE the remote SCON signals of the other CEs are passedthrough an OR circuit connection such as 181 (FIG. 11) to a control linesuch as 120 (FIG. 10).

Other conditions for completing the actuation of AND circuit 176 to setlatch 175 are determined by the signal on line 184 (CCR/CE /bit SC andthe outputs of OR circuits 185 and 186. The output of OR circuit 185 isactive if all of SC bits of the configuration mask CM/CE; conveyed tobus 71 (FIG. 10) are not simultaneously in the 0 state. The output of ORcircuit 186 is active if either AND circuit 187 or AND circuit 170 isenergized or if microprogram control line 189 is excited. AND circuit187 or 170 will be enabled if the values of bits S, and S are bothsimultaneously equal to 1 or simultaneously 0, respectively. Thus, ORcircuit 186 and AND circuits 187 and 170 together comprise an inverseexclusive OR circuit acting upon the outputs of stages S and S of CCR/CBLine 189 is energized during manually initiated loading in an SE of aprogram scheduled for utilization by GE. Hence, this line is identifiedby the symbol IPL (abbreviation for Initial Program Loading Signal).

It should therefore be understood that SCON latch 175 is set to ON ifmicroprogram signal 177 occurs while bit SC, of CCR/CB and at least oneof the bits SC, in CM/CE are coincidentally in a 1 condition and eitherbits 8, and S in CCR/CB are set to identical conditions or control line189 is excited. It should also be understood that latch 175 is reset toOFF by a signal on line 179 invariably following the signal 177. Whilein the ON condition latch 175 controls translation of SM selecting bitsvia line 119, FIG. 10, and via lines corresponding to 120 of FIG. 10 inall other elements.

The handling of mask words CM and SM within the circuits 160 ischaracterized schematically as follows. CM and SM are acquired byprogramming from SE units permitted access to 160 by gates 161. CM andSM are forwarded under microprogram control into selected registers and196 respectively, which are designated R and K; respectively. This isindicated schematically by respective broken lines 197 and 198. CM andSM are parity checked through circuit 162 as indicated by broken lines200 and 201, and are also coupled to outgoing buses corresponding tobuses 71 and 73 shown in FIG. 1. As previously indicated the positionalformat of the CM intelligence is the same for all elements withdifferent bits masked upon entry into different element CCRs. Thus, eachCM contains status bits (S), supervisory control bits (SC), SE bits, CEbits, and IO bits and parity bits in the positions shown in FIG. 6, butthe element CCRs receiving a CM will receive selected portions of a CMand different CMs may be sent out to different elements during areconfiguration program, as will be described below.

The format of SM is indicated at 205. SM is a 36-bit selection controlword including eleven non-selecting bits consisting of four parity bitsP and seven spare bit positions indicated at 207 and 208, andtwenty-five selection control bits which are coupled to individualelement CCR access gates such as AND circuit 101 in FIG. 10, wherebyselective translation to each CCR of a simultaneously issued CM may beeffected. Thus, SM contains six peripheral device adapter selecting bits209, twelve CCR/ SE selecting bits 210, four CCR/CE selecting bits 211,and three CCR/IO selecting bits 212. The values of these bits are 1 forselection and 0 for non-selection of the associated element CCRs.

It would be helpful at this point to consider the program andmicroprogram processes by which a CB can reconfigure the complex.Referring to FIG. 14, this process is characterized in a nine step flowchart in which the sequential sub-program or microprogram steps taken bya CB are numbered 225-233, and discussed below in that order.

Each of the CEs under consideration is designed to operate as anindependent central processing unit (CPU) on instructions of the IBMSystem/360 instruction set. This set of instructions and theirimplementation by a processor are characterized, for example, inco-pending patent application Ser. No. 357,372 of G. M. Amdahl et al.,filed Apr. 6, 1964, which is assigned to the assignee of the presentapplication. Thus, each CE is a processor having internal sets ofcontrols for executing subprograrns or microprograms required forinterpretation and execution of instructions in the above set. It isfurther noted that each CE is provided with additional subprogram ormicroprogram controls for interpreting and executing a SCON instruction,as defined and characterized below, which is not a member of the IBMSystem/ 360 instruction set.

It is assumed that prior to step 225 (FIG. 14) the CCR organization ofthe complex of FIGS. 1-4 is determinable. This means that if the complexis first starting into operation it would be necessary to operate manualreset switches in all elements to establish initial states ofreccptiveness in the CCRs of the CEs appropriate for initiating thereconfiguration sequence defined in FIG. 14. It is assumed that thecomplex is in operation prior to step 225 and a condition has occurredindicating possible need for reorganization.

In step 225 of FIG. 14, CE initiates a microprogram interrupt sequenceof a predetermined class having sensed a signal associated with thatclass of interruption. This may involve sensing of an element checkrequest (ELC) issued by an element other than the sensing CE or aSUPERVISOR (ALL instruction belonging to the above universal set. The CEbegins its interruption sub-routine after completing its last previousoperation. As part of this sub-routine the last instruction address andcertain other items of information are assembled from local registersinto a 64-bit program status word (PSW) which is stored (step 226) at aunique location in an SE with which the CE is then capable ofcommunicating by virtue of the known status of reception controlsettings in the CCR/CE and CCR/SE registers. The unique PSW locationcorresponds to the type or class of interruption.

At step 227 a new PSW is fetched to CE from another unique location inthat SE also corresponding to the class of interruption.

At 228 the new PSW is set up in local buffer registers of the CE and theCE proceeds to recover the first instruction of a new program seriesstored in an SE, in accordance with instruction address intelligencecontained in the new PSW. In this new series the CE is directed (step229) to test the status bits of all elements of the complex, byreferring to a status bit table in a predetermined series of SElocations. This table is shown in FIG. 16 and discussed below withreference to that figure. Depending on the present status of the complexand the particular type of condition which caused the interruption ofstep 225 program branches are taken to direct the CE to either terminatethe current program series (exit 237) and begin an ordinary programsequence or to continue in sequence to address a SCON reconfigurationinstruction located in SE.

Assuming that the sequence leading to step 230 is followed, a SCONinstruction is fetched from SE to local buffers of CE and the PSW in CEis checked to assure that the supervisory mode bit is appropriately set.The SCON instruction has an 18-bit format, parts of which are indicatedat 240. There are eight bits (241) defining an operation code (OP CODE),four bits 242 designating a first local register R in CE, and four hits243 designating 14 a second local register R in CB. Not shown at 240 aretwo parity bits indicating the correct parity of the other sixteen bits.

The registers denoted by R and R are the registers 195 and 196 shown inFIG. 11. These are preset under program control with respective 36-bitwords representing a configuration mask (CM) and selection mask (SM)appropriate to the selected SCON instruction. Such loading may takeplace as part of step 229 or it may have been effected at some previoustime as part of a program process of collecting several CM and SM wordsin several storage locations or registers.

During this phase of the process the SC bits (SC, in the CM word in Rare checked, to make sure that they define a legal configuration, by ORcircuit 185 and AND circuit 176 of FIG. 11. coincidentally, a current SCbit and status bits 8 S in CCR/CE, the SC bits in CM, and the currentstate of the CE test switch (165, FIG. 11) are checked by AND circuits187 and 170, OR circuit 186, and AND circuit 176 (all FIG. 11) to verifythat the CE is in a state in which it is permitted to execute SCON asdiscussed hereafter. If all checks are successful SCON latch (FIG. 11)is set, and simultaneously (231) the CM intelligence accompanied by SMselecting bits is transferred out via appropriate buses (extensions of71, 73, FIG. 11) and presented selectively to the individual CCRreceiving gates of the complex. As each element response is received(232) via bus 250, FIG. 11, the corresponding SM selecting bit in R isreset. Hence, execution of the SCON instruction may be consideredcomplete when all bits in R are set to 0. By storing the SM word inanother register, CE can up-date the CCR status bits of the respondingelements in the above-mentioned status table (step 233).

At this point it is necessary to determine whether the system has beencompletely reconfigured or only partially reconfigured. Hence, it isnecessary to revert CE to state 29 (FIG. 14) and to again conditionallyexit, or proceed to another SCON instruction 231. Eventually, after oneor more passes through loop 229-233 an exit condition is reachedterminating the configuring program.

The status table shown in FIG. 16 includes a pair of current statusbits, S S for each element of the complex, the values of whichcorrespond to conditions stored in the status bit stages of respectiveelement CCRs. To condition program operation 229 on the states of thesehits it is necessary for the programmer to keep in mind the desired endconfiguration of CCR bits and to program towards that end.

A word of explanation is in order at this point. In the exemplaryembodiment under consideration there are five distinct states shown inthe table of FIG. 15, and a programming convention used in the exemplarypresent organization is that all elements in the same state will receiveonly from each other. While this is not an essential restriction itsimplifies the reconfiguration program considerably since theconfiguration mask (CM) for any element can be composed by entering onesin the CM bit positions assigned to other elements in the same state andzeroes in all other bit positions. Without this restriction it would benecessary to program more of the CCR information into the status tableof FIG. 16.

Now let us consider an example as follows. Assume that CB CB SE I0 andall adapters except TCU are in ACTIVE states with their respective pairsof status bits set to the condition 11 (note FIG .16). This means thatall of the real time processing is being handled by these elements, thateach element in this group can receive intelligence from any otherelement of the group to which it has a data path connection, due tocorresponding 1 bits in the CCRs, and that each element in the group isinsensitive, by virtue of 0 bits in its CCR, to communications fromelements not in the group. Now assume CE has failed and it is desired toreassign CE; from the ACTIVE group to the TEST group while bringing inCB presently in REDUNDANT state (5 :1, 5 as a replacement for CE. Thismay be accomplished by means of two SCON instructions executed in twopasses through loop 229233 of FIG. 14, as follows:

CB may execute the reconfiguration program.

The CM word in register R of CE in the first execution of step 231 (FIG.14) would have the form:

The SM word in register R of CE, would have the form:

PAM TCU sscg szp sz cz to have the form:

s so 2-; 52 P2 51: CE 13 IO 11 so our% 00000011 00001001 010 And the SMword for correctly distributing this CM throughout the complex would be:

PAM TQU 1 53 P SE 10 P Assuming that the fault in CE is not located inCCR/ SE or its access network, the above CM will be cor- 'ectly acceptedby CE and the reconfiguration process nay then be terminated. It mightbe mentioned that this assumption is a realistic one since this smallsegment at CE will rarely fail. Upon termination of reconfigura- :ion CEwill be ready to undergo manual tests or autonatic tests under programcontrol of CE When CE 1238 been repaired it will usually be desirable toreconigure CR, or CE; into the REDUNDANT group.

The question of priority relating to simultaneous reuests for service byseveral elements was briefly touched ipon above. It is worthwhile tonote that IO and CE mits will generally not impose significantconflicting re- ;uest loads on an SE since the 10 elements act in re-;ponse to I/O instructions executed by the CEs. A priorty resolutionproblem also occurs when several CEs at- ;empt to share access to one SEsimultaneously or in nterleaved passes. This is a problem only when a CE5 executing an extended program involving many refer- :nces to SE, forexample a data sorting program. It is lot a problem for the procedureinvolved in FIG. 14.

Nevertheless it is noted for the sake of completeness hat hardware isprovided in each SE for selecting a inique access request from among anyplurality of coending requests.

It may be observed that in the example above correipondingly ordered SCand CE bits in the second CM CEP,

word do not have equal values. This results from a spe- :ializedprogramming restriction which is stated as folows: A CM issued to forman ACTIVE or REDUN- DANT group will have SC bits set to l in thepositions corresponding to all ACTIVE and REDUNDANT CEs (this permits aREDUNDANT CE upon receipt of ELC from a CB to reconfigure the complex byfirst resetting the CCR state bits of the receiving CE to the ACTIVEstate (11) as indicated in Note 4 (FIG. 15). A CM issued to form a TESTgroup may contain ls in SC bit positions assigned to CEs in the TESTstate as well as in other positions to permit recall of TEST elements toother states by CEs in other states. The question occurs, what happenswhen an additional non-failing element is required in the TEST group.The answer is that only a CE in the ACTIVE group could safely make sucha reassignment since only such a CE can be presumed to be aware ofcurrent processing needs. It is advisable to add that an element in TESTstate may be less available" to the ACTIVE system than an element inREDUNDANT state since it may be accumulating important test data in amanner not susceptible to interruption, whereas processing in theREDUNDANT group is of the type which is considered instantaneouslyinterruptable; for example, program debugging routines.

It should be obvious that if all CCR bits were stored with the statustable of FIG. 16 considerably more sophisticated reconfigurationschedules could be followed. However, the present plan of organizationsimplifies the reconfiguration programming etIort by limiting the numberof passes through the reconfiguration subroutine to at most four (i.e.the number of distinct states defined by the CCR status bits: ACTIVE(11), REDUNDANT (01), REDUNDANT (10), and TEST (00) and is adequate formost multiprocessing purposes.

CCR/ SE Referring to FIG. 12, CCR/SE, is a register 300 in which thetwelve SE bit positions are unused since one SE cannot communicate withother SEs. The groups of AND circuits 305-308 controlled by outputs ofrespective AND circuits 301-304 are adapted to selectively translaterespective CM words CM/CE CM/CE CM/ CE; and CM/CE to OR circuits 315.AND circuits 301- 304 are individually controlled by SE bits in SM wordsissued by CE respectively, and by bits 5C 8C 8C and SC in CCR/SE(register 300). AND circuits 301- 304 are further jointly controlled bya TEST switch 316 acting through a TEST LATCH 317. With latch 3H7 OFFcircuits 301-304 are each partially enabled. Latch 317 is set OFF whenswitch 316 is OFF and 0N when AND circuit 318 responds to the occurrenceof: TEST switch 316 ON and CCR/SE /bit S =CCR/SE /bit 8 :0.

Manually keyed CCR changes are entered in CCR/SE via ANDs 327,corresponding functionally to ANDs 127 of FIG. 10. OR 333 and bypasscircuit 337 perform functions corresponding to those respectivelyprovided by OR 133 and bypass circuit 137 of FIG. 10. Check circuits notshown provide the parity and other checks relevant to issuance of ELC/SE(on a line also not shown). OR circuit 350 conveys SCON signals from CE2' 3 or 4 via line 320 to ANDs 301-304.

As suggested in FIG. 13, the CCR circuits for a representative IO aresubstantially similar to the CCR/SE circuits of FIG. 12, except that theIO bit locations in CCR/IO are unused. This is because IOs need notexchange information between each other in the present organization. Ofcourse, this is not an essential restriction since it may be desirablein some complexes to permit transfers between IOs.

Referring to FIG. 15, in the subject organization an element can beestablished in any of five distinct states: ACTIVE (8 :1, 8 :1),REDUNDANT (8 :0, 5 :1), REDUNDANT (8 :0, 8 :0), TEST WITH LATCH OFF (8:0, 8 :0), and TEST WITH LATCH ON (S -=0, 8 :0). REDUNDANT-10, differsfrom RE- DUNDANT-01 only in that the former state alfords some

